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Abstract 

Background: The Information Assessment Method (1AM) is a popular tool for continuing education and knowledge 
translation. After a search for information, the 1AM allows the health professional to report what was the search 
objective, its cognitive impact, as well as any use and patient health benefit associated with the retrieved health 
information. In continuing education programs, professionals read health information, rate it using the 1AM, and 
earn continuing education credit for this brief individual reflective learning activity. 1AM items have been iteratively 
developed using literature reviews and qualitative studies. Thus, our research question was: what is the content 
validity of 1AM items from the users' perspective? 

Methods: A two-step content validation study was conducted. In Step 1, we followed a mixed methods research 
design, and assessed the relevance and representativeness of IAM items. In this step, data from a longitudinal 
quantitative study and a qualitative multiple case study involving 40 family physicians were analyzed. In Step 2, 
IAM items were analyzed and modified based on a set of guiding principles by a multi-disciplinary expert panel. 

Results: The content validity of 16 IAM items was supported, and these items were not changed. Nine other items 
were modified. Three new items were added, including two that were extensions of an existing item. 

Conclusion: A content validated version of the IAM (IAM 201 1) is available for the continuing education of health 
professionals. 

Keywords: Content validity, Continuing medical education, Health informatics, Information assessment method, 
Primary health care, Knowledge translation 



Background 

Keeping abreast of medical advances is a challenge, partly 
because of the turnover of research-based information [1], 
In their long-term relationships with patients, family 
physicians (FPs) strive to combine their clinical expertise 
with their patients' preference in arriving at a clinical 
decision [2], From time-to- time, clinical decisions are 
assisted by searches in an Electronic Knowledge Resource 
(EKR) that provide the FP and their patient with the 
research-based health information, or evidence, they seek 
[3]. EKRs can include updated topic summaries, clinical 
practice guidelines, systematic reviews, or synopses. For 
example, Essential Evidence Plus contains databases 
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such as POEMs (Patient-Oriented Evidence that Matters) 
providing access to synopses of new clinical research 
filtered for relevance to primary care. EKRs can meet 
information needs such as: addressing clinical questions, 
supporting decision-making and overcoming the limits 
of human memory [4]. EKRs enable access to evidence 
in a timely way, and help to bridge the gap between 
clinical research and practice, e.g., when they are linked 
to electronic medical records [5]. 

There are questionnaires to evaluate user satisfaction 
with EKRs. However, there is a tool called the Information 
Assessment Method (IAM), which systematically and 
comprehensively assesses the value of information from 
the perspective of the health professional [6]. The IAM 
is a popular tool for continuing education and knowledge 
translation in Canada. About 10,000 physicians and 
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pharmacists have participated in IAM-based continuing 
education programs involving InfoPOEMS and Highlights 
[7-10]. 

It is employed for documenting the reflective learning 
of health professionals [11]. The I AM allows health 
professionals to report the search objective(s), cognitive 
impact, use and patient health benefit associated with 
health information retrieved from electronic knowledge 
resources. These health professionals read health infor- 
mation, rate it using the IAM (brief individual reflective 
learning activity), and earn continuing education 
credits. Our research team at McGill University began 
work on the IAM in 2001 [12]. While IAM items have 
been used for years, they remain to be validated for 
content by the users. Therefore, our research question 
was: what is the content validity of IAM items from the 
users' perspective? The purpose of this paper is to 
present a content validated version of the IAM, which 
we will call IAM 2011. 

The IAM can concurrently assess the following: reasons 
for an information search, the cognitive impact of retrieved 
health information, any use of that information, and 
subsequent expected patient health benefits. When linked 
to uniquely identified objects of health information, the 
IAM is a systematic and comprehensive user-centered 
method to assess the value of information [6]. In its 
current form, in the context of information retrieval, 
the IAM questionnaire contains 26 items linked to four 
constructs: (a) search objective, (b) cognitive impact, 
(c) the use of information for a specific patient, and (d) 
information related patient health benefit. 

The IAM is based on the Acquisition-Cognition-Ap- 
plication-Outcome (ACAO) theoretical model, which we 
have derived from the work of Saracevic and Kantor in 
information science [13]. Our ACAO model sequentially 
integrates the intention to search for health information, 
related cognitive impacts, corresponding use of this infor- 
mation, and patient health benefit [14]. 

Acquisition is the process of getting information or 
objects potentially conveying information, as related 
to some intentions. Cognition is the process of ab- 
sorbing, understanding, integrating the information. 
Application is the process of use (using) this newly 
understood and cognitively processed information. Out- 
come is the specific end-result from applying this 
information. 

Content validation 

Content validity is defined as "the degree to which ele- 
ments of an assessment instrument are relevant to and 
representative of the targeted construct for a particular 
assessment purpose" [15]. Content validation of an 
assessment instrument sometimes involves refinement. 
The decision to refine elements or items of an instrument 



or develop new items depends on knowing which are 
performing poorly. Items might be considered for deletion 
or modification only if the facets of the targeted construct 
are not compromised [16]. 

In measurement, a construct refers to the attributes 
or variables that are targets of assessment [15]. Thus, 
each component that the IAM assesses is a construct, e. 
g., acquisition (search objective). Constructs are com- 
posed of facets, which aid in assessing the construct. 
The term factor is synonymously used with the term 
facets. We chose not to use the term 'factor', because it 
leads the reader to think of a category of data analytic 
techniques, such as factor analysis [17]. The items of 
our IAM questionnaire reflect the facets of each target 
construct. For each construct, the development of facets 
was based on literature reviews and empirical studies 
[6]. We carried out a mixed studies review to (a) iden- 
tify whether the facets of the four target constructs of 
the IAM are currently mentioned (either present or 
absent) in the literature, and (b) identify any additional 
facets that the IAM does not currently include. The four 
constructs and their respective facets are presented 
in Table 1. 

Figure 1 depicts our citation tracking search strategy. 
A deductive-inductive thematic analysis was performed 
on 73 included studies that used quantitative, qualitative 
or mixed methods. We coded relevant excerpts from 
these studies to corresponding themes belonging to 
IAM facets. The identification of new themes suggested 
new facets. Thus, all IAM facets tested below were sup- 
ported by at least one included study. 

Methods 

A mixed methods research study was conducted. Mixed 
methods research is defined as a combination of quantitative 
and qualitative methods [18]. Indeed, content validation is a 
mixed methods process that is applicable to all elements of 
an assessment instrument [15]. Content validity is composed 
of relevance and representativeness [15]. Relevance is a 
measure of the appropriateness of the items of an instru- 
ment to assess a target construct. Hence, relevant items 
indicate the essential facets in assessing the target 
construct. Representativeness refers to the extent to 
which the elements represent the facet to be assessed. 
Since relevance is considered a measure, quantitative 
methods were used to evaluate the relevance of IAM 
items. Representativeness reflected the extent to which an 
item clearly represented the facet that was being assessed, 
hence qualitative methods were used to assess the repre- 
sentativeness of IAM items. 

Design 

Figure 2 represents our mixed methods convergence 
design. The convergence design obtains different but 
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Table 1 Information assessment method: target 
constructs and facets 



complimentary data on the same case [19]. The conver- 
gence design enables separate data collection and analysis 
of the same phenomenon, and subsequently the different 



results are converged during the interpretation stage, 
by comparing and contrasting quantitative and qualitative 
results. 

Setting, participants and intervention 

Forty Canadian FPs were recruited through personal con- 
tacts at various medical meetings or through e-newsletters 
distributed by the College of Family Physicians of Canada. 
All participants were offered a handheld computer at 
no cost, for participating in this study. In exchange for 
participation, FPs were also offered continuing medical 
education credits for each search for health information 
they rated. All participants were practicing FPs. The 
average age of participants was 44 years; there were 17 
women and 23 men. Twenty-eight (68%) had a connection 
through teaching or research to a faculty of medicine. 
Participants agreed to rate their searches in Essential 
Evidence Plus, using the IAM questionnaire. Both Es- 
sential Evidence Plus and the IAM software were provided 
on their handheld device. 

Quantitative data collection/analysis 

A longitudinal observational study was conducted, whereby 
participants searched for information on their handheld 
computer in routine clinical practice over an average of 
320 days. Searches for information contained various 
information objects, such as synopses of clinical research 
(POEMs), guidelines and abstracts of Cochrane reviews. 
When a participant opened one information object and 
responded to the IAM questionnaire, it was defined as a 
rated hit'. The IAM software generated log reports which 
included: (a) date and time of all information searches that 
included rated 'hits; (b) titles of rated hits (the information 
objects), and (c) IAM item responses by physician linked 
to specific 'hits'. The IAM item responses were collected 
at the search level (search objective) and at the 'hit' level 
(cognitive impact and use). The IAM item responses 
form the pool of quantitative data. A total of 1,767 rated 
searches were performed by the participants during the 
study period which comprised 3,300 rated 'hits'. Quan- 
titative data analysis for each of the four constructs: Item 
relevance (R) was calculated using the following formula. 

Number of the items was rated or explained 
Total number of rating or explanations 

For items where R < 10%, we considered the relevance 
of that item to be questionable. We chose R < 10% as the 
cut-off for questioning item relevance for two reasons: 
(a) there is no agreed upon criterion or universal cut off 
to determine the extent of content validity [20]; and (b) 
since the IAM items are based on 10 years of research 
in a focused context, it was likely that many items would 
be relevant. Hence, in order to identify items that had 



Target construct: 
Acquisition 



Target construct: 
Cognition 



Target construct: 
Application 



Target construct: 
Outcome 



Facets: Types of search objectives 

(1) Address a clinical question/problem/decision- 
making about a specific patient 

(2) Fulfill an educational or research objective 

(3) Search in general or for curiosity 

(4) Look up something I had forgotten 

(5) Share information with a patient/ caregiver 

(6) Exchange information with other health 
professionals 

(7) Plan, manage, coordinate, delegate or monitor 
tasks with other health professionals 

Facets: Types of cognitive impacts 

(1) My practice was (will be) changed and 
improved 

(2) I learned something new 

(3) This information confirmed I did (I am doing) 
the right thing 

(4) I was reassured 

(5) I recalled something 

(6) I was dissatisfied, as this information had no 
impact on my practice 

(7) I was dissatisfied as there was a problem 
with this information 

(8) I disagree with this information 

(9) I think this information is potentially harmful 

Facets: Types of use of information for a 
specific patient 

(1) To modify the management of this patient 

(2) To justify or maintain the management 
of this patient 

(3) To better understand a particular issue 
related to this patient 

(4) To persuade other health professionals or 
patients to make changes 

Facets: Types of patient health benefit 

(1) Increasing patient knowledge about 
heath or healthcare 

(2) Avoiding unnecessary or inappropriate 
treatment, diagnostic procedure or 
preventative intervention 

(3) Increasing patient acceptability of treatment, 
diagnostic procedure or preventative 
intervention 

(4) Preventing disease or health deterioration 
(including acute episodes of chronic diseases) 

(5) Improving patient health or functioning or 
resilience (i.e., how well the patient faces 
difficulties) 
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LOOP 1:3 
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References 
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studies 
"who cited 
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64 potentially 
relevant studies 




Citation 
tracking 

References 
cited by the 
relevant 
studies 
"who they 
cited" 



References 
n = 72 



References 
excluded 
based on full 
text 
n = 63 



Nine new potentially 
relevant studies were found 
at the end of Loop 1 



LOOP 2: Nine new 

studies from Loop 
1 



Citation 
tracking 

References 
citing the 
relevant 
studies 
"who cited 
them" 





83 

references 
retrieved 





Nine potentially relevant studies 
were found at the end of Loop 1 



Citation 
tracking 

References 
cited by the 
relevant 
studies 
"who they 
cited" 



References 
excluded 
based on 
abstract / 
title 
n = 81 



References 
excluded 
based on full 
text 
n = 2 



Figure 1 Flow chart depicting the stages of our literature review. The nine new references that were found in Loop 1 were used to initiate Loop 2. 
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Quantitative Data 




Quantitative Analysis 





Log Reports: 
Tracked searches for 
each physician. No. 
of physicians for 
every IAM item 
checked. 



Descriptive 
Statistics: 
RELEVANCE 



Quantitative 
Results 



Qualitative Analysis 




Qualitative Data 





Inductive-deductive 
thematic analysis: 
REPRESENTATIV- 
ENESS 



Transcripts of 
individual interviews 
with physicians 



Qualitative 
Results 



Integration : ^ 
Interpreting 
Content Validity 



Figure 2 Visual iagram of the mixed methods study design. 



very low relevance we chose the arbitrary cutoff of 10%. 
For example, the number of times the item "Address a 
clinical question" was checked = 1310; the total number 
of times any of the seven items in the acquisition con- 
struct (Search Objectives) was checked = 4253; therefore, 
R for this item was x 100 = 31% and this item was 
deemed to be relevant. 

Qualitative data collection/analysis 

A multiple case study was conducted, where a case was 
defined as one rated search submitted by one participant. 
For these rated searches, participants were interviewed to 
explain a purposeful sample of their rated information 
'hits'. The emphasis was on 'purposively selecting informa- 
tion-rich' cases [21]. As a result, the purposeful sample 
consisted of 'hits' where information was used for a 
specific patient or where information had a negative 
type of cognitive impact. A participants explanation for 
checking one IAM item, for one information 'hit' was 
defined as a 'unit'. 

The interviewer was an anthropologist experienced in 
conducting interviews. Interviews were conducted based 
on a semi-structured questionnaire (See Additional file 1). 
Individual log reports containing a list of rated searches 
were e-mailed to each FP as an aide memoire, before the 
interview. The interviews were recorded with permission 
of each participant. Most interviews were conducted by 
telephone and lasted 30 to 45 minutes. Each participant 
was interviewed twice during the data collection phase. 

During the interview, participants were reminded of their 
rated search for information using their IAM item re- 
sponses, in terms of: 'Search objectives' and 'Cognitive im- 
pacts'. The participants provided explanations as to why 
they checked a particular IAM item during a search for 
information. Additionally, open questions sought explana- 
tions as to how the information was used for a specific pa- 
tient (if applicable) and to describe any patient health 
benefits (perceived or witnessed during follow-up contact). 



If the participants described any information use or patient 
health benefit further questions were asked using a list of 
items of information use for a specific patient and patient 
health benefit, following an interview guide. 

Qualitative data analysis was used to evaluate repre- 
sentativeness of IAM items. To facilitate this analysis we 
used specialized software (NVivo7). An inductive-de- 
ductive thematic analysis of the interviews was carried 
out using IAM items as initial themes [22]. Each unit or 
a physician's explanation for one rated IAM item was 
coded to the respective theme based on the IAM item 
definitions (presented in Additional file 2). There were 
four possibilities: (1) a unit was a 'FIT' when the physi- 
cian's explanation was concordant with the definition of 
the IAM item, (2) 'MISFIT' when it was concordant with 
the definition of another IAM item, (3) 'UNCLEAR; when 
an explanation was provided, but this explanation was not 
clear, (4) 'NEW when an explanation did not correspond 
to any of the current IAM item definitions and was a po- 
tential new item, and (5) 'NONE; when no explanation 
was provided. We coded a total of 3,199 units during the 
thematic analysis process. An item was considered 
representative if the number of 'FIT' units was > 80% of all 
responses (FIT + MISFIT + UNCLEAR + NEW + NONE). 
There is no agreed upon criterion for determining the ex- 
tent to which a measure has attained content validity. 
Nunnally [23] noted that "inevitably content validity 
rests mainly on appeals to reason regarding the adequacy 
with which important content has been cast in the form of 
test items." 

Integration of quantitative and qualitative components, 
and expert panel discussion 

The integration of the quantitative and qualitative methods 
occurred at the stage of interpreting quantitative (rele- 
vance) and qualitative (representativeness) results. After 
this integration, we modified problematic items based 
on a list of recommendations on how to compose good 
questionnaire items [24-26]. We subjected a draft 
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version of the proposed I AM 2011 for review by a 
panel of seven persons comprised of FP researchers, li- 
brarians, information scientists and an anthropologist 
This was a multidisciplinary panel of researchers with ex- 
pertise in studying the value of information. Expert panel 
discussion is a core component of Content Validation 
Guidelines [15]. Nunnally and Bernstein [27] also noted 
that results from such an exercise can guide judgments 
about the content validity of any items that need modi- 
fication or need to be omitted. The panel members 
evaluated and suggested changes for each item given its 
relevance to the construct, representativeness to the 
item definition, clarity, language, and response format. 

Based on opinions from this expert panel, final changes 
were made, and the IAM 2011 questionnaire was created. 

Results 

The relevance and representativeness for each IAM item 
are presented in Table 2. Sixteen items were both rele- 
vant and representative and were retained without modi- 
fication. For example, item: To look up something I had 
forgotten (Relevance =16% and Representativeness = 88%). 
Nine items were relevant but not representative and were 
considered for modification. For example, item: / recalled 
something (Relevance =18% and Representativeness = 
78%). From our qualitative data analysis we found that 
participants understood this item as being reminded of 
something that they already knew' or had seen before. 
Hence we suggested a modification of this item. Three 
items were found to be representative but not relevant. 
For example, item: / disagree with this information 
(Relevance = 4% and Representativeness = 66%). We sug- 
gested removal of the items that were not relevant. We 
also found one new theme/facet that was suggested by our 
qualitative data: The use of information in a discussion 
with this patient or other health professionals. This facet of 
information use was not present in previous versions of 
the IAM, while it was reported 53 times in 30 searches 
for information. 

Example 

Interviewer: "Would you say this information would have 
any consequences for the patient?" 

MD25: "Well, it might, / would probably have a discus- 
sion with the patient about the therapeutic options if the 
diagnosis was confirmed, so, you know. (...) they would 
probably have more information before going ahead to 
meet a specialist (...)." 

The Expert Panel analyzed the results of our mixed 
methods study and decided to modify eight items; this 
led to the final version of IAM 2011. 

For example, consider the following old item: There was 
a problem with this information. Experts commented 
that the wording of this item might overlap with 



'dissatisfaction'. Furthermore, the old item does not 
emphasize problems with respect to structure and 
amount of information. Thus we modified the item as 
follows u There is a problem with the presentation of this 
information". 

The panel also suggested that one double barreled item 
needed to be split into two items. Old item: To justify or 
maintain the management of this patient. This item was 
split into two: (a) I hesitated between options for this 
patient, and I used (will use) this information to justify 
a choice, and (b) I did not know what to do, and I used 
this information to manage this patient. Detailed results 
are available online [28] . 

Discussion 

The results from our mixed methods study and Expert 
Panel Discussion have led us to propose a content vali- 
dated version of the IAM containing 28 items (seven for 
Search Objectives, nine for Cognitive Impacts, seven for 
Information Use for a Specific Patient, and five for Patient 
Health Benefit) (Figure 3). This mixed methods study 
helped us identify items that needed modification (Repre- 
sentativeness < 80%) or removal (Relevance < 10%). 

Based on the decisions of our expert panel, item modifi- 
cations were made. However, three items that were consid- 
ered for possible removal were retained, for the following 
reasons: 

Item: Plan, manage, coordinate, delegate or monitor tasks 
with other health professionals 

In about 1,800 rated searches for information, 4,253 
reasons were selected by participants, from which this 
item was endorsed only 197 times. This suggests its 
relevance is approximately 5%. Although this item was 
highly representative, participants did not commonly rate 
this item as a reason for searching for health information. 
Participants rated this item if they were teaching or 
supervising a resident or when they had to collaborate 
with a nurse, pharmacist, respiratory therapist, subspecialist 
(e.g., rheumatologist), or a patients family doctor. We 
considered the following two possible causes for the 
low relevance of this item: (a) the EKR that was used in 
the study might have contained little information on 
collaborating with other health professionals around 
patient care, hence, the frequency of rating this item 
was low; and (b) the item included more than one reason 
to search (plan or manage or coordinate or delegate), 
which is not a best practice. Taking both possible issues 
into consideration, we retained the item, but with the 
following modification - "To manage aspects of patient 
care with other professionals". 

Item: "1 disagree with this information" and Item: "1 think 
this information is potentially harmful" 

These two items of negative cognitive impact had low 
relevance (4% and 8%). This implies that participating 
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Table 2 Relevance and representativeness of 1AM items 

Items 

Reasons for information search 

1. Address a clinical question/problem/decision-making about a specific | 

2. Fulfill an educational or research objective 

3. Search in general or for curiosity 

4. Look up something I had forgotten 

5. Share information with a patient/ caregiver 

6. Exchange information with other health professionals 

7. Plan, manage, coordinate, delegate or monitor tasks with other health 
professionals 

Cognitive impact 

Items of Positive Impact 

1. My practice was (will be) changed and improved 

2. I learned something new 

3. This information confirmed I did (I am doing) the right thing. 

4. I was reassured 

5. I recalled something 
Items of Negative Impact 

6. 1 was dissatisfied as this information had no impact on my practice 
7. 1 was dissatisfied as there was a problem with this information 
8. 1 disagree with this information 
9. I think this information is potentially harmful 

Information use for a specific patient 

1 . To modify the management of this patient 

2. To justify or maintain the management of this patient 

3. To better understand a particular issue related to this patient 



Relevant? 


Representativ? 


Decision 


Number of 






ratings = 4253 






YES 


YES 


Retain 


31% 


98% 




YES 


YES 


Retain 


10% 
YES 


98% 
YES 


Retain 


12% 


12% 




YES 


YES 


Retain 


16% 
YES 


88% 
YES 


Retain 


15% 


93% 




YES 


YES 


Retain 


12% 


97% 




NO 


YES 


Consider removal 


5% 


86% 




Number of ratings = 






6329 






YES 


YES 


Retain 


1 CO/, 


QQO/, 
CO 70 




YES 


YES 


Retain 


jU /O 
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YES 


YES 


Retain 


24% 
YES 


88% 
YES 


Retain 


23% 


90% 




YES 


NO 


Consider 




78% 


modification 


18% 




Number of 






ratings = 166 






YES 


YES 


Retain 


47% 


83% 




YES 


YES 


Retain 


40% 


83% 




NO 


NO 


V— \j 1 1 jiuci i ci i iuvai 




DD70 




NO 


YES 


Consider removal 


8% 


80% 




Number of units = 737 






YES 


NO 


Consider modificati 


19% 


53% 




YES 


YES 


Retain 


39% 


92% 




YES 


YES 


Retain 


28% 


97% 
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Table 2 Relevance and representativeness of 1AM items (Continued) 



1 U pcljUdUc Ulllcl lltrdllll pi Ul coolUI Idlo Ul Udllclllo LU 1 1 Idlxc LI Idl lytro 


YES 


NO 


V_UI lolUtri 






/y/0 


modification 




1 AWn 

I /o 


rcHieru neann ucneTit 
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1. Increasing patient knowledge about heath or healthcare 


T LJ> 


VF<; 

I LJ 


Retain 




Zj/0 


yo /o 




2. Avoiding unnecessary or inappropriate treatment, diagnostic procedure 


YES 


YES 


Retain 


or preventative intervention 


21% 


88% 




3. Increasing patient acceptability of treatment, diagnostic procedure or 


YES 


NO 


Consider 


preventative intervention 


18% 


3% 


modification 


4. Preventing disease or health deterioration (including acute episodes 


YES 


NO 


Consider 


of chronic diseases) 


17% 


64% 


modification 


5. Improving patient health or functioning or resilience (i.e., how well the 


YES 


NO 


Consider 


patient faces difficulties) 


20% 


66% 


modification 



FPs rarely disagreed with the information they retrieved 
nor did they believe it was harmful. One explanation for 
this result is that participants used an EKR (Essential 
Evidence Plus) that contained information filtered for 
validity and relevance. Thus, harmful or problematic 
information was rarely identified. Although these two 
items were not relevant in this study, they can potentially 
contribute to research on the value of health information 
and help identify harmful information in other EKRs. 
Thus we decided to retain these two items of negative 
cognitive impact. 

Our study faced a limitation: the limits of human mem- 
ory. The number of days between a search for information 
on the handheld computer and telephone interviews 
varied widely, from 1 to 250. This issue was addressed 
by excluding searches that participants could not clearly 
remember. It is possible that the value of forgotten 
searches may be different from those that were not for- 
gotten. In addition, participants represented a convenience 
sample of 40 FPs and their item ratings may be different 
from those obtained from a larger random sample of FPs. 
However, a sufficiently large sample of 3,300 rated hits 
was obtained for the purpose of content validation. More- 
over, a representative sample of the targeted population is 
not needed for content validity assessment. For example, 
according to Vogt et al. [29], three to six focus groups with 
five to 10 participants in each group are sufficient. 

The strength of our mixed methods content validation 
study is that the members of the target population (FPs) 
were consulted as experts when they used the IAM in 
their routine clinical practice. Participants had the 
opportunity to use the IAM questionnaire over a long 
period of time to provide a perspective on IAM items. 
This longitudinal approach is preferred over a focus 



group discussion that would have provided only a 
snapshot of the collective opinion [29]. 

In future educational research, the IAM can be used 
to compare EKRs, e.g., Essential Evidence Plus versus 
DynaMed. The data collected through IAM 2011 can 
be used for assessing the value of research-based 
health information from a users' perspective. Finally, 
content validity is an integral component of construct val- 
idity, and construct validity is the degree to which an 
assessment instrument measures the targeted construct 
[15]. Future research should examine the construct valid- 
ity of IAM 2011 for EKRs, e.g., using factor analysis 
techniques. 

Conclusion 

The content validated IAM 2011 can systematically and 
comprehensively document reflective learning in medical 
education, and facilitate the evaluation of education pro- 
grams. In other words, the IAM is both an intervention at 
the individual level, and a program evaluation tool at the 
collective level [30]. At the participant level (individual 
response), reading information and assessing it with IAM 
qualifies as a 'brief individual reflective learning' activity, 
which allows the provision of continuing education 
credits. The IAM is currently being used in various 
Canadian continuing medical education programs, e.g., 
in one program, physicians are asked to read and rate 
their searches for information in databases such as 
DynaMed. At the program level (all responses from all 
participants), IAM documents three educational out- 
comes (participation, learning, performance), while 
many current program evaluations provide only docu- 
mentation of attendance. 
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Q4. Did you (will you) use this information for a specific patient? 

Yes O 
No O 

Instruction to programmers: If 'No', subsequent items and question 5 are disabled. 

If Yes, please check all that apply. You can check more than one type of use. Yes No 

As a result of this information I managed (or will manage) this patient differently O O 

I hesitated between options for this patient, and I used (will use) this information to justify a choice O O 
I did not know what to do, and I used (will use) this information to manage this patient O O 

I thought I knew what to do, and I used this information to be more certain about the management O O 
of this patient 

I used this information to better understand a particular issue related to this patient O O 

I used (will use) this information in a discussion with this patient, or with other health professionals O O 
about this patient 

I used (will use) this information to persuade this patient, or to persuade other health professionals O O 
to make a change for this patient 

Q5. For this patient, did you observe (do you expect) any health benefits as a result of applying this information? 

Yes O 
No O 

Instruction to programmers: If 'No', subsequent items are disabled. 

If Yes, please check all that apply. You can check more than one type of benefit. Yes No 

This information helped to improve (will help to improve) this patient's health status, functioning or O O 
resilience (i.e., the ability to adapt to significant life stressors) 

This information helped to prevent (will help to prevent) a disease or worsening of disease for this O O 
patient 

This information helped to avoid (will help to avoid) unnecessary or inappropriate treatment, O O 

diagnostic procedures, preventative interventions or referral to another specialist, for this patient 
This information helped to decrease this patient's worries about a treatment, diagnostic procedure O O 
or preventative intervention 

This information helped to increase this patient's knowledge, or their family or home health aides' O O 
knowledge 

Comment on this information or this questionnaire. 



Figure 3 Content validated version of the information assessment method (1AM 201 1) 
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